Deploy Prometheus Using Helm Chart in K8S

Recently, when deploying Prometheus, I found it surprisingly difficult to find a good tutorial on k8s - based deployment online. Even the tutorial in the official repo of the helm chart is not user - friendly. Many parameters are not explained in detail, and there are a lack of examples.

I recommend two resource addresses that I think are quite good.

1. https://prometheus.wang/
   This resource claims to be the Chinese documentation of Prometheus.
   Although the content is not extremely extensive, it comprehensively introduces the usage tutorials of Prometheus and its related tools, from deployment and installation to practical usage examples. You should be able to go through all the content in it in one day. Personally, I think it’s a great introductory tutorial. It won’t make you proficient in Prometheus, but it’s sufficient for getting started. Moreover, it’s in Chinese, which is very friendly to us.

2. https://prometheus.io/docs/introduction/overview/
   This is the address of the official website tutorial. It is recommended that you first go through the Chinese tutorial above, and then read the official website tutorial as needed. Don’t try to read the documentation from start to finish right away. It’s a waste of time and quite difficult (it’s in English, and it’s hard to truly understand many parts without practical experience).

3. [https://github.com/prometheus - community/helm - charts](https://github.com/prometheus - community/helm - charts)
   This is the official repo of the Prometheus helm chart. Although the content is scarce, I still recommend it. After all, it’s official. I hope they can spend more time on the documentation in the future.

Deployment steps:
First of all, you need to have a k8s cluster and be able to use helm normally. The other steps are as follows:

# Add the helm chart library
helm repo add prometheus - community https://prometheus - community.github.io/helm - charts

# Export the configurable part of the configuration file. This file needs to be modified with parameters that meet your needs.
helm show values prometheus - community/prometheus >> values.yaml

# After modifying the default configuration file in values.yaml, you can deploy.
helm install prometheus --namespace prometheus -f values.yaml prometheus - community/prometheus

The reason why I said the documentation in the official repo is poor is that they don’t have a very detailed description and example configuration for the configurable values.yaml (that is, out - of - the - box usability).

I’ve placed the configuration I used below for reference only.

The following points need to be noted:

1. storageclass
   It is recommended to directly provide a storageclass and let helm create the PVC by itself, rather than creating the PVC yourself. This is more convenient. Also, when creating the storageclass, pay attention to adding the parameter allowVolumeExpansion: true. Otherwise, the PVC cannot change its capacity. It will be troublesome when the initially created capacity of the PVC is used up in the future because it cannot be expanded.
   In AWS, EFS can actually be used as the backend storage of the storageclass because EFS has unlimited capacity, so there is no need to worry about the expansion problem.

2. configmaps (configuration files)
   When deploying Prometheus using the helm chart, the configuration files of the Prometheus server and alertmanager are mounted to the pod through a configmap. Therefore, in the server and alertmanager sections of values.yaml, there is a parameter “configMapOverrideName” that allows you to configure your own configmap. If you don’t configure this parameter, it will generate one by itself.

I suggest leaving this parameter empty and letting the helm chart generate it. Then, use kubectl to export the generated configmap, modify the configuration, and apply it. This is easier for beginners.

# Export the configmap and modify it
kubectl get configmaps -n prometheus <configmap - name> -o yaml > configmap.yaml
# Apply the modified configmap
kubectl apply -f configmap.yaml

The following are the values.yaml parameters I actually used:

rbac:
  create: true

podSecurityPolicy:
  enabled: false

imagePullSecrets:
# - name: "image - pull - secret"

## Define serviceAccount names for components. Defaults to component's fully qualified name.
##
serviceAccounts:
  alertmanager:
    create: true
    name:
    annotations: {}
  nodeExporter:
    create: true
    name:
    annotations: {}
  pushgateway:
    create: true
    name:
    annotations: {}
  server:
    create: true
    name:
    annotations: {}

alertmanager:
  ## If false, alertmanager will not be installed
  ##
  enabled: true

  ## Use a ClusterRole (and ClusterRoleBinding)
  ## - If set to false - we define a Role and RoleBinding in the defined namespaces ONLY
  ## This makes alertmanager work - for users who do not have ClusterAdmin privs, but wants alertmanager to operate on their own namespaces, instead of clusterwide.
  useClusterRole: true

  ## Set to a rolename to use existing role - skipping role creating - but still doing serviceaccount and rolebinding to the rolename set here.
  useExistingRole: false

  ## alertmanager container name
  ##
  name: alertmanager

  ## alertmanager container image
  ##
  image:
    repository: quay.io/prometheus/alertmanager
    tag: v0.21.0
    pullPolicy: IfNotPresent

  ## alertmanager priorityClassName
  ##
  priorityClassName: ""

  ## Additional alertmanager container arguments
  ##
  extraArgs: {}

  ## Additional InitContainers to initialize the pod
  ##
  extraInitContainers: []

  ## The URL prefix at which the container can be accessed. Useful in the case the '-web.external - url' includes a slug
  ## so that the various internal URLs are still able to access as they are in the default case.
  ## (Optional)
  prefixURL: ""

  ## External URL which can access alertmanager
  baseURL: "http://localhost:9093"

  ## Additional alertmanager container environment variable
  ## For instance to add a http_proxy
  ##
  extraEnv: {}

  ## Additional alertmanager Secret mounts
  # Defines additional mounts with secrets. Secrets must be manually created in the namespace.
  extraSecretMounts: []
    # - name: secret - files
    #   mountPath: /etc/secrets
    #   subPath: ""
    #   secretName: alertmanager - secret - files
    #   readOnly: true

  ## ConfigMap override where fullname is {{.Release.Name}}-{{.Values.alertmanager.configMapOverrideName}}
  ## Defining configMapOverrideName will cause templates/alertmanager - configmap.yaml
  ## to NOT generate a ConfigMap resource
  ##
  configMapOverrideName: ""

  ## The name of a secret in the same kubernetes namespace which contains the Alertmanager config
  ## Defining configFromSecret will cause templates/alertmanager - configmap.yaml
  ## to NOT generate a ConfigMap resource
  ##
  configFromSecret: ""

  ## The configuration file name to be loaded to alertmanager
  ## Must match the key within configuration loaded from ConfigMap/Secret
  ##
  configFileName: alertmanager.yml

  ingress:
    ## If true, alertmanager Ingress will be created
    ##
    enabled: true

    ## alertmanager Ingress annotations
    ##
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/affinity - mode: "persistent"
      nginx.ingress.kubernetes.io/session - cookie - name: "route"
    #   kubernetes.io/tls - acme: 'true'

    ## alertmanager Ingress additional labels
    ##
    extraLabels: {}

    ## alertmanager Ingress hostnames with optional path
    ## Must be provided if Ingress is enabled
    ##
    hosts: 
      - alertmanager - stg.example.com
    #   - domain.com/alertmanager

    ## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
    extraPaths: []
    # - path: /*
    #   backend:
    #     serviceName: ssl - redirect
    #     servicePort: use - annotation

    ## alertmanager Ingress TLS configuration
    ## Secrets must be manually created in the namespace
    ##
    tls: []
    #   - secretName: prometheus - alerts - tls
    #     hosts:
    #       - alertmanager.domain.com

  ## Alertmanager Deployment Strategy type
  # strategy:
  #   type: Recreate

  ## Node tolerations for alertmanager scheduling to nodes with taints
  ## Ref: https://kubernetes.io/docs/concepts/configuration/assign - pod - node/
  ##
  tolerations: []
    # - key: "key"
    #   operator: "Equal|Exists"
    #   value: "value"
    #   effect: "NoSchedule|PreferNoSchedule|NoExecute(1.6 only)"

  ## Node labels for alertmanager pod assignment
  ## Ref: https://kubernetes.io/docs/user - guide/node - selection/
  ##
  nodeSelector: {}

  ## Pod affinity
  ##
  affinity: {}

  ## PodDisruptionBudget settings
  ## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
  ##
  podDisruptionBudget:
    enabled: false
    maxUnavailable: 1

  ## Use an alternate scheduler, e.g. "stork".
  ## ref: https://kubernetes.io/docs/tasks/administer - cluster/configure - multiple - schedulers/
  ##
  # schedulerName:

  persistentVolume:
    ## If true, alertmanager will create/use a Persistent Volume Claim
    ## If false, use emptyDir
    ##
    enabled: true

    ## alertmanager data Persistent Volume access modes
    ## Must match those of existing PV or dynamic provisioner
    ## Ref: http://kubernetes.io/docs/user - guide/persistent - volumes/
    ##
    accessModes:
      # - ReadWriteOnce
      - ReadWriteOnce

    ## alertmanager data Persistent Volume Claim annotations
    ##
    annotations: {}

    ## alertmanager data Persistent Volume existing claim name
    ## Requires alertmanager.persistentVolume.enabled: true
    ## If defined, PVC must be created manually before volume will be bound
    existingClaim: ""

    ## alertmanager data Persistent Volume mount root path
    ##
    mountPath: /data

    ## alertmanager data Persistent Volume size
    ##
    size: 50Gi

    ## alertmanager data Persistent Volume Storage Class
    ## If defined, storageClassName: <storageClass>
    ## If set to "-", storageClassName: "", which disables dynamic provisioning
    ## If undefined (the default) or set to null, no storageClassName spec is
    ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
    ##   GKE, AWS & OpenStack)
    ##
    storageClass: "prometheus - storageclass"

    ## alertmanager data Persistent Volume Binding Mode
    ## If defined, volumeBindingMode: <volumeBindingMode>
    ## If undefined (the default) or set to null, no volumeBindingMode spec is
    ##   set, choosing the default mode.
    ##
    # volumeBindingMode: ""

    ## Subdirectory of alertmanager data Persistent Volume to mount
    ## Useful if the volume's root directory is not empty
    ##
    subPath: ""

  emptyDir:
    ## alertmanager emptyDir volume size limit
    ##
    sizeLimit: ""

  ## Annotations to be added to alertmanager pods
  ##
  podAnnotations: {}
    ## Tell prometheus to use a specific set of alertmanager pods
    ## instead of all alertmanager pods found in the same namespace
    ## Useful if you deploy multiple releases within the same namespace
    ##
    ## prometheus.io/probe: alertmanager - teamA

  ## Labels to be added to Prometheus AlertManager pods
  ##
  podLabels: {}

  ## Specify if a Pod Security Policy for node - exporter must be created
  ## Ref: https://kubernetes.io/docs/concepts/policy/pod - security - policy/
  ##
  podSecurityPolicy:
    annotations: {}
      ## Specify pod annotations
      ## Ref: https://kubernetes.io/docs/concepts/policy/pod - security - policy/#apparmor
      ## Ref: https://kubernetes.io/docs/concepts/policy/pod - security - policy/#seccomp
      ## Ref: https://kubernetes.io/docs/concepts/policy/pod - security - policy/#sysctl
      ##
      # seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
      # seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
      # apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'

  ## Use a StatefulSet if replicaCount needs to be greater than 1 (see below)
  ##
  replicaCount: 2

  ## Annotations to be added to deployment
  ##
  deploymentAnnotations: {}

  statefulSet:
    ## If true, use a statefulset instead of a deployment for pod management.
    ## This allows to scale replicas to more than 1 pod
    ##
    enabled: true

    annotations: {}
    labels: {}
    podManagementPolicy: OrderedReady

    ## Alertmanager headless service to use for the statefulset
    ##
    headless:
      annotations: {}
      labels: {}

      ## Enabling peer mesh service end points for enabling the HA alert manager
      ## Ref: https://github.com/prometheus/alertmanager/blob/master/README.md
      enableMeshPeer: false

      servicePort: 80

  ## alertmanager resource requests and limits
  ## Ref: http://kubernetes.io/docs/user - guide/compute - resources/
  ##
  resources:
    limits:
      cpu: 200m
      memory: 1024Mi
    requests:
      cpu: 20m
      memory: 100Mi

  # Custom DNS configuration to be added to alertmanager pods
  dnsConfig: {}
    # nameservers:
    #   - 1.2.3.4
    # searches:
    #   - ns1.svc.cluster - domain.example
    #   - my.dns.search.suffix
    # options:
    #   - name: ndots
    #     value: "2"
  #   - name: edns0

  ## Security context to be added to alertmanager pods
  ##
  securityContext:
    runAsUser: 65534
    runAsNonRoot: true
    runAsGroup: 65534
    fsGroup: 65534

  service:
    annotations: {}
    labels: {}
    clusterIP: ""

    ## Enabling peer mesh service end points for enabling the HA alert manager
    ## Ref: https://github.com/prometheus/alertmanager/blob/master/README.md
    # enableMeshPeer : true

    ## List of IP addresses at which the alertmanager service is available
    ## Ref: https://kubernetes.io/docs/user - guide/services/#external - ips
    ##
    externalIPs: []

    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    servicePort: 80
    # nodePort: 30000
    sessionAffinity: None
    type: ClusterIP

## Monitors ConfigMap changes and POSTs to a URL
## Ref: https://github.com/jimmidyson/configmap - reload
##
configmapReload:
  prometheus:
    ## If false, the configmap - reload container will not be deployed
    ##
    enabled: true

    ## configmap - reload container name
    ##
    name: configmap - reload

    ## configmap - reload container image
    ##
    image:
      repository: jimmidyson/configmap - reload
      tag: v0.5.0
      pullPolicy: IfNotPresent

    ## Additional configmap - reload container arguments
    ##
    extraArgs: {}
    ## Additional configmap - reload volume directories
    ##
    extraVolumeDirs: []


    ## Additional configmap - reload mounts
    ##
    extraConfigmapMounts: []
      # - name: prometheus - alerts
      #   mountPath: /etc/alerts.d
      #   subPath: ""
      #   configMap: prometheus - alerts
      #   readOnly: true


    ## configmap - reload resource requests and limits
    ## Ref: http://kubernetes.io/docs/user - guide/compute - resources/
    ##
    resources: {}
  alertmanager:
    ## If false, the configmap - reload container will not be deployed
    ##
    enabled: true

    ## configmap - reload container name
    ##
    name: configmap - reload

    ## configmap - reload container image
    ##
    image:
      repository: jimmidyson/configmap - reload
      tag: v0.5.0
      pullPolicy: IfNotPresent

    ## Additional configmap - reload container arguments
    ##
    extraArgs